The Role of Models in Predictive Validation

Model choice and validation have a central role in data analysis, including predictive modeling. While standard diagnostics can help identify model inadequacies, it is natural to use predictive accuracy as the decisive criterion in the final choice of predictive model. A key point is that any assessment of predictive accuracy, theoretical or empirical, inevitably assumes a “data mechanism”, i.e., a sampling or other stochastic model that relates model predictions to the population that is the target for predictions. In a controversial paper Breiman [3] presents predictive accuracy as an obvious and natural criterion for model assessment. He criticises a statistical culture that has almost exclusively used models that assume a stochastic “data mechanism” that is thought to describe underlying scientific processes. Breiman argues for the wider use of algorithmic models, such as tree-based regression and neural nets, that “treat the data mechanism as unknown”. Disregard of data mechanisms has limits. Simple approaches to predictive validation assume, in effect, that the data are a random sample from the population to which predictions will be applied. This is often inappropriate. Cox [3, following comment] notes that predictions are often applied “under quite different conditions from the data”. Indeed, the conditions may be so different that a realistic assessment of predictive accuracy is impossible! In what follows I will (1) comment briefly on algorithmic models; (2) note that the interest is, in some contexts, in model parameters; (3) comment in more detail on predictive accuracy; (4) discuss implications for data mining and for the use of data bases.